Interpretation of Meaningful Expressions by Integrating Gesture and Posture Modalities

Integration of different modalities aims to increase the robustness of a system and improves its performance in an unconstrained environment. With this motivation, this research is focused on integrating two modalities such as hand gesture and posture in a unified framework where the criterion is measured by employing Particle Filter system. The proposed framework has two main modules: 1) Gesture and posture recognition system, and 2) Particle filter based integration of these systems by incorporating CFG rules. In first module, the gesture and posture features are extracted and classified by HMM and SVM respectively. In the second module, the integration is carried by mapping the classification outcome on Particle filter system which acts as contribution-weight at decision level. Moreover, to infer and extract the “meaningful expressions” from the input sequence based on their contribution-weights, we have exploited regular grammar and developed the production rules using context free grammar (CFG) for our test scenario (i.e. a restaurant). Experiments are conducted on 500 different combinations of restaurant orders with the overall 98.3% inference accuracy whereas the classification accuracy of gesture and posture recognition is approximately 98.6% which proves the significance of proposed approach.